Customer support solution recommendation system

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer-readable storage medium and a method for automatically providing support solutions in response to user feedback items. The method comprises receiving user feedback items and corresponding support solutions. The method further comprises identifying, using clustering techniques, associations between the user feedback items and the corresponding support solutions. The method further comprises storing the identified associations as an items-solutions model that correlates the user feedback items with the corresponding support solutions. The method further comprises receiving a new user feedback item. The method further comprises automatically determining, using the items-solutions model, at least one support solution that corresponds to the new user feedback item. The method further comprises providing the at least one support solution in response to the received new user feedback item.

BACKGROUND

This specification relates to information presentation.

The Internet provides access for users to a wide variety of resources.For example, video and/or audio files, as well as web pages forparticular subjects or particular news articles, are accessible over theInternet. Additionally, users who are content sponsors may use theInternet to access resources such as advertising interfaces or other websites. If a user encounters a problem (e.g., “cannot login.” “lostpassword,” etc.) while using a resource, the user can generate and sendan email or some other form of user feedback communication regarding theissue to a support center associated with the resource. The supportcenter may receive thousands or millions of user feedback items during atypical month or year. Many of the user feedback items can be related,or may be the exact same issue, and many of the responses (or solutions)by the support center may be similarly related.

SUMMARY

In general, one innovative aspect of the subject matter described inthis specification can be implemented in methods that include a methodfor automatically providing support solutions in response to userfeedback items. The method comprises receiving user feedback items andcorresponding support solutions. The method further comprisesidentifying, using clustering techniques, associations between the userfeedback items and the corresponding support solutions wherein usingclustering techniques includes filtering a corpus of user feedback itemsagainst a domain-specific dictionary. The method further comprisesstoring the identified associations as an items-solutions model thatcorrelates the user feedback items with the corresponding supportsolutions. The method further comprises receiving a new user feedbackitem. The method further comprises automatically determining, using theitems-solutions model, at least one support solution that corresponds tothe new user feedback item. The method further comprises providing theat least one support solution in response to the received new userfeedback item.

These and other implementations can each optionally include one or moreof the following features. Storing the identified associations canfurther comprise identifying groups of related user feedback itemssharing a support solution. Identifying groups of related user feedbackitems sharing a support solution can include applying a term frequencyanalysis to the user feedback items and the support solutions to developa mapping between user feedback items and one or more best-mappedsupport solutions. The support solutions can comprise predeterminedresponses. The predetermined responses can be organized by product andissue. The predetermined responses can represent a library of mostfrequently presented solutions to customer problems presented in theuser feedback items. One or more support solutions can comprise atemplatized message to a customer, describing a step-by-step solution toa specific customer problem. The templatized message can be an emailmessage. The templatized message can comprise one or more placeholdersto be filled based on attributes of a specific customer. The method canfurther comprise receiving a new user feedback item, automaticallydetermining a solution to the new user feedback item using theitems-solutions model, and providing the determined solution responsiveto the new user feedback item. The method can further comprise receivinga new user feedback item, automatically determining one or more proposedsolutions to the new user feedback item using the items-solutions model,and providing the determined one or more proposed solutions.Automatically determining the one or more proposed solutions can includedetermining a confidence score associated with each of the one or moreproposed solutions and ranking the one or more proposed solutions bytheir confidence scores, where providing the determined one or moreproposed solutions includes providing the one or more proposed solutionsthat have highest-ranked confidence scores. Providing the determined oneor more proposed solutions can include providing, to a user, adescription of the one or more proposed solutions; receiving, from theuser, a selection of at least one solution from the one or more proposedsolutions; and providing, in response to the received new user feedbackitem, the user-selected at least one solution.

In general, another innovative aspect of the subject matter described inthis specification can be implemented in systems that include a systemfor automatically providing support solutions in response to userfeedback items. The system comprises a category engine for producingdomain-specific categories using a corpus of user feedback items and adomain-specific dictionary. The system further comprises an associationengine for receiving user feedback items and corresponding supportsolutions; identifying, using clustering techniques and thedomain-specific categories, associations between the user feedback itemsand the corresponding support solutions; and storing the identifiedassociations as an items-solutions model that correlates the userfeedback items with the corresponding support solutions. The systemfurther comprises a solution engine for receiving a new user feedbackitem; automatically determining, using the items-solutions model, atleast one support solution that corresponds to the new user feedbackitem; and providing the at least one support solution in response to thereceived new user feedback item.

In general, another innovative aspect of the subject matter described inthis specification can be implemented in computer program products thatinclude a computer program product tangibly embodied in acomputer-readable storage device and comprising instructions that, whenexecuted by a processor, perform a method for providing content. Themethod comprises receiving user feedback items and corresponding supportsolutions. The method further comprises identifying, using clusteringtechniques, associations between the user feedback items and thecorresponding support solutions. The method further comprises storingthe identified associations as an items-solutions model that correlatesthe user feedback items with the corresponding support solutions. Themethod further comprises receiving a new user feedback item. The methodfurther comprises automatically determining, using the items-solutionsmodel, at least one support solution that corresponds to the new userfeedback item. The method further comprises providing the at least onesupport solution in response to the received new user feedback item.

These and other implementations can each optionally include one or moreof the following features. Storing the identified associations canfurther comprise identifying groups of related user feedback itemssharing a support solution. Identifying groups of related user feedbackitems sharing a support solution can include applying a term frequencyanalysis to the user feedback items and the support solutions to developa mapping between user feedback items and one or more best-mappedsupport solutions. The support solutions can comprise predeterminedresponses. The predetermined responses can be organized by product andissue. The predetermined responses can represent a library of mostfrequently presented solutions to customer problems presented in theuser feedback items.

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features, aspects, andadvantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example environment for providing supportsolutions in response to user feedback items.

FIG. 2 is a flow chart of an example process for creating anitems-solutions model and using the model to automatically determinesolutions.

FIG. 3 is a flow chart of an example process for using a domain-specificfeature set to categorize feedback items into categories.

FIG. 4 is a block diagram of example computing devices that may be usedto implement the systems and methods described in this document.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

This document describes systems, methods and techniques forautomatically recommending support solutions for user feedback itemswithin environments having large numbers of user feedback items. Forexample, an online support center (e.g., a web site that handlescustomer problems for an online product or service) can receive millionsof items of user/customer feedback (e.g., problems, issues, questions,comments, etc.) per month over a variety of different communicationchannels. Example communication channels include e-mail, chat, helpforums, messaging platforms, message boards, postings, dislikeindications, or telephone calls. In some implementations, automatedprocesses can be used to organize and understand the corresponding highlevel of volume of customer feedback items. Some implementations caninclude a number of automated, machine learning-based methods thatcluster and categorize customer feedback and the correspondingsolutions. Clustering techniques can include filtering a corpus of userfeedback items against a domain-specific dictionary. For example,clusters can be used to aggregate solutions by groups of related issues.Information provided by the clusters can then be used, for example, toautomatically identify a solution for a new issue using solutions thatpreviously have been used to solve instances of the same or similarissues.

In some implementations, automatically providing solutions cancompletely eliminate or significantly reduce the need for humaninvolvement in the solution, e.g., by requiring little or no action bycustomer support, help desk or other support personnel to providesolutions. In some implementations, some customer queries may still needto be manually processed and answered, e.g., by customer supportrepresentatives (CSRs). The proposed systems and methods can be used tosupport the CSRs. For example, the CSRs can select one or more solutionsfrom an automatically-generated set of proposed or recommendedsolutions. The CSR can select one or more appropriate solutions from theautomatically —generated list and forward the solutions to the customer.

In some implementations, the selection of clusters can depend oncategories of domain-specific features (e.g., words and phrases).Example domains include online advertising systems, online stores,auction houses, and so on. In some implementations, domain-specificfeatures can be determined by filtering a corpus of user feedback itemsagainst a domain-specific dictionary. For example, to create categoriesof domain-specific features (e.g., “ad,” “campaign,” “creative,”“keyword,” etc.) for an online advertising system, the customer feedbackfor the online advertising system can be filtered using a dictionary ofterms (e.g., key terminology items) specific to online advertisingsystems. In some implementations, such domain-specific dictionaries canbe compiled automatically from product documentation including, forexample, help center documents, user documentation, a corpus of pastuser communications (e.g., email), and so on.

Another example source of documents for compiling a domain specificdictionary is a repository of auto-respond e-mails, e.g., containingpredefined solutions for most frequently-occurring customer problems.This can be a good source of product-specific terminology becausepredefined solutions usually contain key terminology for solvingcustomer problems that are specific to a given product.

Some implementations can use an additional or alternative model ofcustomer support by phone. For example, an asynchronous model can beused in which a customer can leave a voice message describing theirproblem or question. Then, later (e.g., within the hour or that sameday), the customer can receive personalized help over e-mail. SMS, orsome other preferred channel of communication. The personalized help canbe generated, for example, after automatically processing the voicemessage.

For example, in one sequence of events, instead of making the customerwait on hold (e.g., to talk to a live person), the call center systemcan offer an option to leave a voice message. In some implementations,the option can include triggering a recording (e.g., played to thecustomer) that prompts the customer to include customer identificationand a brief description of the problem or question. In someimplementations, the system can use speech recognition software totranscribe the customer's voice message into text.

In some implementations, unsupervised and supervised machine learningclassification techniques can be used to process the text created fromthe voice message in order to classify the problem or question. Forexample, the system can automatically identify the specific productissue the customer is inquiring about, as well as assign a numericdegree of confidence that is associated with the classification of theissue (e.g., 85% certain that the problem involves “Logging into ProductX”). In some implementations, e.g., using supervised machine leaning, auser (e.g., call center personnel) can provide input into the processsuch as to review and select one or more of the automatically-generatedclassifications.

Continuing with the example, based on the classification determined forthe customer issue, and provided that the corresponding degree ofconfidence of the classification is above a predefined quality threshold(e.g. 80% or higher), the system can automatically recommend potentialsolutions to the customer. For example, the proposed solution(s) can bedelivered back to the customer over e-mail, SMS text, or some otherpreferred method.

In some implementations in which the voice message can be received inany one of multiple languages, language detection can be performedautomatically by the speech recognition system. For example, if acustomer in Paris leaves a message in French, the speech recognitionsystem can detect French as the language. In some implementations, thespeech recognition system can generate text in French and automaticallyconvert the text to English in order to help to identify the correctclassification.

FIG. 1 is a diagram of an example environment 100 for providing supportsolutions in response to user feedback items. The environment 100includes user devices 102 (e.g., from which the user feedback itemsoriginate) that are communicatively connected, through a network 104,with support systems 106 and an items-solutions model 108. The userdevices 102, for example, can be used by users who access onlineresources, such as resources that are supported by the support systems106. The user devices 102 can also host web browsers (e.g., by whichusers can access the resources) and other applications (e.g., email) forsubmitting user feedback items. The support systems 106 can include oneor more online service/support entities, such as online advertisingsystems, online stores, auction houses, etc. The network 104 can includeany combination of the Internet, local area networks (LANs), wide areanetworks (WANs), and other networks. Interactions that occur between theuser devices 102 and the support systems 106, including informationabout support solutions used for particular user feedback items, can beused to generate information that is stored in the items-solutions model108.

In some implementations, the items-solutions model 108 can include datastores of past user feedback items 110 and support solutions 112 which,in combination, can be used by an association engine 114 to produce andstore associations 116. For example, the association engine 114 can useclustering techniques to create clusters in the associations 116 thatinclude clusters representing groups of related user feedback items 110that share support solutions 112. A solution engine 118 can then use theassociations 116 to automatically generate one or more solutions for anew user feedback item that is processed by the items-solutions model108.

In some implementations, the support solutions 112 can includepredetermined responses (e.g., organized by product and issue) thatrepresent a library or repository of most frequently-presented solutionsto customer problems presented in the user feedback items. For example,one or more predetermined responses can exist for login/password issuesrelated to a particular online advertising system.

In some implementations, the one or more support solutions can include atemplatized message to a customer, describing a step-by-step solution toa specific customer problem. For example, a templatized message forpassword/login problems can include multiple steps or points toconsider. A template that is used in this example can have the commontext, such as an email form, that includes text and/or other contentthat is common to all instances of the particular problem. The templatecan also include placeholders for particular fields and information inthe form that can then be filled in as part of the response, e.g., basedon attributes of the specific customer and the particular problem. Forexample, the placeholders can be used to fill in the user's name,particular information for the current instance of the password/loginproblem, identification of the product (e.g., “Example AdvertisingSystem”), and so on. In some implementations, template messages can bein the form of an email message or some other message format (e.g., anSMS text, a voice message, etc.)

In some implementations, a category engine 119 can producedomain-specific categories 120. For example, a domain can comprise oneor more advertising system products, and the feature set can includewords and phrases that are specific to the advertising system productsdomain. By using the categories, the associations 116 that are producedby the association engine 114 can be more focused to the domain.

In one example flow shown in FIG. 1, any number of user devices 102 canprovide user feedback items 121 (stage 1). This can occur continuouslyover time, e.g., as each user submits a user feedback item 121 for theirparticular problem or issue (e.g., with an online advertising system).For example, the users can use email or other interfaces on the userdevices 102 to submit their user feedback items 121. The support system106 (e.g., for the online advertising system) can receive the userfeedback items 121 and generate one or more support solutions 122 (stage2). The support system 106 can provide the support solutions 122 to theparticular user device 102 that submitted the particular user feedbackitem 121 to which the support solutions 122 apply.

At the same time or thereafter, the items-solutions model 108 canreceive user feedback items and corresponding support solutions 124 forstorage as user feedback items 110 and support solutions 112 (stage 3).Further, as indicated by connection line 126, information stored in theuser feedback items 110 and/or the support solutions 112 can includeinformation that identifies what support solution(s) correspond to eachuser feedback item. Stages 1-3 can repeat several times (e.g., hundreds,thousands or more) over time as additional user feedback items 121 arepresented, and support solutions 122 are provided in response.

Continuing with the current example flow, the association engine 114 canuse clustering techniques to build clusters representing associationsamong related user feedback items 110 (stage 4). For example, theassociation engine 114 can store the clusters and any other information(e.g., including information identifying the solutions 112 thatcorrespond to the user feedback items 110) in the associations 116.Conventional clustering techniques can be used to cluster the userfeedback items. In some implementations, the clustering techniques canutilize a high-quality text clustering algorithm, such as thedistributed exchange algorithm, in which the clustering can produce anarrangement of user feedback items that have the maximum number ofkeywords in common within each cluster, and therefore a high probabilityof describing the same specific topic.

In some implementations, clusters representing associations amongrelated user feedback items can also effectively cluster thecorresponding solutions of user feedback items. This can occur, forexample, because proposed solutions for a user feedback item are oftenembedded in the e-mail threads as predetermined responses, e.g., aslinks to help center articles, help forum posts, etc. In someimplementations, multiple solutions for a given problem can be clusteredtogether as a result of e-mail thread clustering. In someimplementations, the clusters can be used to enhance the items-solutionsmodel 108, e.g., with one-to-many links from a user feedback item tomultiple possible solutions. This can facilitate the selection of a bestsolution among a variety of possible solutions for a given problem. Insome implementations, the possible solutions can be ranked based on thefrequency of their occurrence, or based on other signals, such ascustomer satisfaction surveys. In some implementations, more than onesolution can be offered to the customer, e.g., with a recommendation totry them in sequence, in case the first one does not work.

At some later time, such as after the items-solutions model 108 ispopulated, a user device 102 can generate a new user feedback item 128(stage 5). In some implementations, the support systems 106 can receivethe new user feedback item 128 and automatically access theitems-solutions model 108 to attempt to identify a solution thatcorresponds to the new user feedback item 128. In some implementations,the solution engine 118 can access the associations 116 and provide oneor more automatically-generated support solutions 130 to the user device102 (stage 6).

In some implementations, each support system 106 can host its ownitems-solutions model 108. For example, the user feedback items 110 andsupport solutions 112 can be local to the support system 106, which canupdate each with user feedback items 121 that are received and thesupport solutions 122 that are generated in response. Moreover, thesolution engine 118, which can also be local to the support system 106,can be invoked for each user feedback item 121 received in order toautomatically generate recommended solutions based on associationsbetween past user feedback items and resulting support solutions.

FIG. 2 is a flow chart of an example process 200 for creating anitems-solutions model and using the model to automatically determinesolutions. The process 200 includes stages that can be implemented asinstructions and executed by one or more processors in one or morecomputer systems. In some implementations, the process 200 can beperformed by the support systems 106, the association engine 114, andthe solution engine 118. FIG. 1 is used for examples for the stages ofthe process 200.

User feedback items and corresponding support solutions are received(202). As an example, the support system 106 can receive user feedbackitems 121 from the user devices 102. The user feedback items 121 caninclude, over time, many (e.g., hundreds, thousands or more) instancesof problems including “cannot login,” “lost password,” etc. In response,the support systems can generate support solutions 122, which can besent to the user devices 102. Support solutions 122 can includesolutions, for example, for each of the problems that include “cannotlogin.” “lost password,” and so on. At the same time or thereafter, theitems-solutions model 108 can receive the user feedback items andcorresponding support solutions 124 for storage as user feedback items110 and support solutions 112.

Using clustering techniques that include filtering a corpus of userfeedback items against a domain-specific dictionary, associationsbetween the user feedback items and the corresponding support solutionsare identified (204). For example, the association engine 114 canidentify associations (e.g., using clustering techniques) using the userfeedback items 110 and support solutions 112 and store the associations116. The clustering techniques, for example, can include filtering acorpus of user feedback items, such as emails, etc., against adomain-specific dictionary. The domain-specific dictionary, for example,can be derived from documents that correspond to a domain (e.g., callcenter manuals for an online advertising system), including a search ofthe documents to identify most commonly-occurring words or phrases. Theassociations can identify groups of related user feedback items thatshare common support solutions. Details of clustering techniques aredescribed below with reference to FIG. 3. Example results of theclustering can include combining user issues such as “cannot login,”“lost password” and others into one or more clusters because theproblems and/or the corresponding solutions are related. For example,user feedback issues related to “cannot login,” “lost password” and“password expired” can be included in a cluster corresponding to “cannotlogin”/“lost password” or “login issues.”

In some implementations, identifying the associations can includeapplying a term frequency analysis to the user feedback items and thesupport solutions to develop a mapping between user feedback items andone or more best-mapped support solutions. For example, for a group ofuser feedback items 110 that include issues such as “cannot login” and“lost password,” there can be one or more support solutions 112 thatsolve those issues. Examples solutions can include inquiring whether“Caps Lock” is on, providing steps for re-setting a password, providingpassword hints, and so on.

The identified associations are stored as an items-solutions model thatcorrelates the user feedback items with the corresponding supportsolutions (206). As an example, the association engine 114 can store theassociations in the data store of associations 116. In someimplementations, the associations can be stored as clusters.

A new user feedback item is received (208). As an example, after theitems-solutions model 108 is generated, the support system 106 canreceive a new user feedback item 128, such as a problem with adescription of “cannot login.” “lost password,” “how do I modifytargeting keywords?” or some other issue.

At least one support solution that corresponds to the new user feedbackitem is automatically determined using the items-solutions model (210).For example, using a description such as “cannot login” for a new userfeedback item 128, the solution engine 118 can look up the one or moreproposed solutions corresponding to the cluster “login problems” todetermine one or more solutions.

In some implementations, determining the one or more proposed solutionscan include determining a confidence score associated with each of theone or more proposed solutions. The one or more proposed solutions canbe ranked by their confidence scores, and the proposed solutions havingthe highest-ranked confidence scores can be selected. For example, aconfidence score of 89 can represent an 89% chance that the proposedsolution will succeed in correcting the problem. The solution engine118, for example, may identify several possible solutions when accessingthe associations 116 to identify a solution for a particular userfeedback item 121. As an example, if ten possible solutions areidentified, the solution engine 118 can assign confidence scores to eachof the ten possible solutions that are based, for example, on the numberof times that each of the possible solutions was used previously tosolve problems identical or similar to the user feedback item 121. Insome implementations, the solution engine 118 can select thehighest-ranked solutions that were used at least a threshold number oftimes (e.g., at least a hundred). As a result, less then all (e.g., fiveor six) of the ten possible solutions can be selected.

In some implementations, a confidence score of a recommended solutioncan be based on the confidence of the classification of the userfeedback item. In implementations using unsupervised clustering, theconfidence score can correspond to a purity metric of the cluster. Inimplementations using classification-based clustering, the confidencescore can correspond to a numeric classification confidence valueprovided by the classifier.

In some implementations, confidence scores can be provided to customersupport representatives (CSRs) who can use the information to help inselecting from proposed solutions in order to identify solutions toprovide to the user. In some implementations, confidence scores can beprovided with the solutions that are presented to the user. For example,templatized predetermined responses can include “We think your problemwill be solved by X (90% chance), but if not try Y and Z”.

The at least one support solution is provided in response to thereceived new user feedback item (212). For example, the solution engine118 can provide the one or more solutions (e.g., corresponding to thecluster “login problems.”) to the support system 106 and/or the userdevice 102.

In some implementations, when a high confidence score exists that one ormore particular solutions are applicable to a particular user feedbackitem, the solution engine 118 can provide the solution directly to theuser device 102. In some implementations, the solution engine 118 canprovide the solution to the support system 106 which can forward thesolutions to the user device 102. In some implementations, a supportuser (e.g., a CSR) can review the one or more recommended solutionsprovided by the solution engine 118 and select one or more of thesolutions (and/or provide some other solutions not recommended) toforward to the user device 102.

FIG. 3 is a flow chart of an example process 300 for using adomain-specific feature set to categorize feedback items intocategories. The process 300 includes stages that can be implemented asinstructions and executed by one or more processors in one or morecomputer systems. In some implementations, the process 300 can beperformed by the association engine 114. FIG. 1 is used for examples forthe stages of the process 300.

Feedback items received from one or more users are identified (302). Asan example, the identified feedback items can be the user feedback items110 that are collected over time within the items-solutions model 108.In some implementations, the support systems 106 can store user feedbackitems 121 that are received from user devices 102.

A domain associated with the feedback items is identified, and a featureset is identified for the domain (304). One example domain can compriseone or more advertising system products, and the feature set can bespecific to the advertising system products domain. The feature set inthis example can include words and phrases that are specific to theadvertising system products domain, e.g., that are contained incorresponding user feedback items. In some implementations, one or moredomains can be associated with a single support system 106.

In some implementations, identifying the feature set can includeidentifying statistically important words or phrases using a termfrequency-inverse document frequency (TF-IDF) analysis of the pluralityof feedback items. For example, the TF-IDF score that is assigned to aword or phrase can be higher depending on how often the word or phraseappears is the corpus of feedback items. However, stop words such as“the” and “and,” which may appear in virtually every feedback item, canbe assigned a lower TF-IDF score.

In some implementations, identifying the feature set can include usingone or more whitelists to identify words or phrases to include in thefeature set. For example, the whitelist for the domain associated withan online advertising service can include words such as “ad,”“campaign,” “creative,” “keyword,” and so on.

In some implementations, the one or more whitelists can be generatedautomatically by analyzing domain-related documents. For example,computer applications can analyze a corpus of documents that correspondto a domain (e.g., call center manuals for an online advertisingsystem), including searching the documents to identify mostcommonly-occurring words or phrases. The words or phrases that are notstop words can be added to the whitelist, e.g., discarding overlyfrequently-occurring words (e.g., “and,” “the,” etc.) and keeping theremaining words or phrases that have high frequencies.

In some implementations, identifying the feature set includes filteringwords and phrases from the feature set to remove undesirable words orphrases using one or more blacklists. For example, the one or moreblacklists can include one or more stop words (e.g., “the,” “and,”etc.), greetings (e.g., “Hello,” “Dear,” etc.), selected proper names(e.g., “Smith.” “Jones” or any other names of people, agents, or CSRstaff), common email boilerplate text (e.g., “from,” “subject,”“sincerely,” etc.), email headers and footers, normalized UniversalResource Locators (URLs), monetary terms, help center page links, phonenumbers, addresses, etc.

In some implementations, identifying the feature set includes evaluatingeach feedback item to determine words or phrases to include in thefeature set and disregarding pre-determined portions of feedback itemsbased on regular expressions. For example, the disregarded portions ofthe feedback items include one or more of title, headers, or footersthat can be detected using techniques involving regular expressions.

In some implementations, identifying the feature set includesidentifying a sample of item responses that relate to the feedback itemsand extracting one or more terms for the feature set from the itemresponses. For example, a small percentage (e.g., 1% or less) of supportsolutions 112 can be analyzed, and frequently-occurring terms can beextracted. The feature set can be based on terms occurring with a highfrequency (e.g., above a threshold), while disregarding terms that occurtoo frequently, such as stop words.

Using the feature set, the feedback items are categorized intocategories (306). For example, the category engine 119 can cluster thefeedback items into categories, where each category includes one or moreclusters of related user feedback items. One example category for “loginissues” can include user feedback issues related to “cannot login,”“lost password,” “password expired” and so on.

In some implementations, categorizing the feedback items can includeusing a machine learning system to categorize the feedback items. Forexample, unsupervised algorithms, including clustering algorithms, candetect similarities among related feedback items, including termrelative frequencies and distributions.

In some implementations, categorizing can include determining categoriesbased on one or more of a volume of feedback items, a number of uniquepost threads associated with the feedback items, a number of uniqueusers who have posted the feedback items, and/or a number of uniqueaddresses associated with the feedback items. For situations in whichthe systems discussed here collect personal information about users, theusers may be provided with an opportunity to opt in/out of programs orfeatures that may collect personal information (e.g., information abouta user's preferences or a user's current location). User informationthat is used to identify unique users, unique addresses or otheruser-related history can be anonymized so that the privacy of users isprotected. For example, quasi-unique identifiers can be associated withusers, but the actual identifying information of the users is notstored. Additionally, any identified user interactions can begeneralized (for example, generalized based on user demographics) ratherthan associated with a particular user. Encryption and obfuscationtechniques can also be used to protect the privacy of users.

The categories are stored (308). For example, the category engine 119can store the categories in the categories 120.

In some implementations, the categorized feedback items can be providedto assigned response providers. For example, categories associated withadvertising system products can be provided to support systems 106 thatare associated with online advertising systems.

In some implementations, as new user feedback items are received overtime, a determination can be made whether the received new user feedbackitems are associated with a new category that has not been stored (e.g.,in the categories 120). Over time, a growth in volume of the receivednew user feedback items can be monitored. When the growth in volumeexceeds a pre-determined threshold (e.g., N number of instances within atime period T), the category engine 119 can categorize the received userfeedback items into a new category, and the category engine 119 canstore the new category in the categories 120.

FIG. 4 is a block diagram of computing devices 400, 450 that may be usedto implement the systems and methods described in this document, aseither a client or as a server or plurality of servers. Computing device400 is intended to represent various forms of digital computers, such aslaptops, desktops, workstations, personal digital assistants, servers,blade servers, mainframes, and other appropriate computers. Computingdevice 450 is intended to represent various forms of mobile devices,such as personal digital assistants, cellular telephones, smartphones,and other similar computing devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexemplary only, and are not meant to limit implementations of theinventions described and/or claimed in this document.

Computing device 400 includes a processor 402, memory 404, a storagedevice 406, a high-speed interface 408 connecting to memory 404 andhigh-speed expansion ports 410, and a low speed interface 412 connectingto low speed bus 414 and storage device 406. Each of the components 402,404, 406, 408, 410, and 412, are interconnected using various busses,and may be mounted on a common motherboard or in other manners asappropriate. The processor 402 can process instructions for executionwithin the computing device 400, including instructions stored in thememory 404 or on the storage device 406 to display graphical informationfor a GUI on an external input/output device, such as display 416coupled to high speed interface 408. In some implementations, multipleprocessors and/or multiple buses may be used, as appropriate, along withmultiple memories and types of memory. Also, multiple computing devices400 may be connected, with each device providing portions of thenecessary operations (e.g., as a server bank, a group of blade servers,or a multi-processor system).

The memory 404 stores information within the computing device 400. Insome implementations, the memory 404 is a computer-readable medium. Insome implementations, the memory 404 is a volatile memory unit or units.In other implementations, the memory 404 is a non-volatile memory unitor units.

The storage device 406 is capable of providing mass storage for thecomputing device 400. In some implementations, the storage device 406 isa computer-readable medium. In various different implementations, thestorage device 406 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In some implementations,a computer program product is tangibly embodied in an informationcarrier. The computer program product contains instructions that, whenexecuted, perform one or more methods, such as those described above.The information carrier is a computer- or machine-readable medium, suchas the memory 404, the storage device 406, or memory on processor 402.

The high-speed controller 408 manages bandwidth-intensive operations forthe computing device 400, while the low speed controller 412 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 408is coupled to memory 404, display 416 (e.g., through a graphicsprocessor or accelerator), and to high-speed expansion ports 410, whichmay accept various expansion cards (not shown). In some implementations,low-speed controller 412 is coupled to storage device 406 and low-speedexpansion port 414. The low-speed expansion port, which may includevarious communication ports (e.g., USB, Bluetooth™, Ethernet, wirelessEthernet) may be coupled to one or more input/output devices, such as akeyboard, a pointing device, a scanner, or a networking device such as aswitch or router, e.g., through a network adapter.

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 420, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 424. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 422. Alternatively, components from computing device 400 may becombined with other components in a mobile device (not shown), such asdevice 450. Each of such devices may contain one or more of computingdevice 400, 450, and an entire system may be made up of multiplecomputing devices 400, 450 communicating with each other.

Computing device 450 includes a processor 452, memory 464, and aninput/output device such as a display 454, a communication interface466, and a transceiver 468, among other components. The device 450 mayalso be provided with a storage device, such as a Microdrive or otherdevice, to provide additional storage. Each of the components 450, 452,464, 454, 466, and 468, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 452 can process instructions for execution within thecomputing device 450, including instructions stored in the memory 464.The processor may also include separate analog and digital processors.The processor may provide, for example, for coordination of the othercomponents of the device 450, such as control of user interfaces,applications run by device 450, and wireless communication by device450.

Processor 452 may communicate with a user through control interface 458and display interface 456 coupled to a display 454. The display 454 maybe, for example, a TFT LCD display or an OLED display, or otherappropriate display technology. The display interface 456 may compriseappropriate circuitry for driving the display 454 to present graphicaland other information to a user. The control interface 458 may receivecommands from a user and convert them for submission to the processor452. In addition, an external interface 462 may be provided incommunication with processor 452, so as to enable near areacommunication of device 450 with other devices. External interface 462may provide, for example, for wired communication (e.g., via a dockingprocedure) or for wireless communication (e.g., via Bluetooth or othersuch technologies).

The memory 464 stores information within the computing device 450. Insome implementations, the memory 464 is a computer-readable medium. Insome implementations, the memory 464 is a volatile memory unit or units.In some implementations, the memory 464 is a non-volatile memory unit orunits. Expansion memory 474 may also be provided and connected to device450 through expansion interface 472, which may include, for example, aSIM card interface. Such expansion memory 474 may provide extra storagespace for device 450, or may also store applications or otherinformation for device 450. Specifically, expansion memory 474 mayinclude instructions to carry out or supplement the processes describedabove, and may include secure information also. Thus, for example,expansion memory 474 may be provide as a security module for device 450,and may be programmed with instructions that permit secure use of device450. In addition, secure applications may be provided via the SIM cards,along with additional information, such as placing identifyinginformation on the SIM card in a non-hack able manner.

The memory may include for example, flash memory and/or MRAM memory, asdiscussed below. In some implementations, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 464, expansionmemory 474, or memory on processor 452.

Device 450 may communicate wirelessly through communication interface466, which may include digital signal processing circuitry wherenecessary. Communication interface 466 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 468. In addition, short-range communication may occur, suchas using a Bluetooth, Wi-Fi, or other such transceiver (not shown). Inaddition, GPS receiver module 470 may provide additional wireless datato device 450, which may be used as appropriate by applications runningon device 450.

Device 450 may also communicate audibly using audio codec 460, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec 460 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device 450. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device 450.

The computing device 450 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 480. It may also be implemented as part of asmartphone 482, personal digital assistant, or other similar mobiledevice.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. While this specification contains manyspecific implementation details, these should not be construed aslimitations on the scope of any inventions or of what may be claimed,but rather as descriptions of features specific to particularimplementations of particular inventions. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking andparallel processing may be advantageous.

What is claimed is:
 1. (canceled)
 2. A method, comprising: receiving,through an audio codec of a personal digital assistant, spoken userinput; automatically determining, by one or more processors, using anitems-solutions model and based on a phrase included in the spoken userinput, one or more support solution responses to the spoken user input,wherein the one or more processors generate the items-solution modelprior to receiving the spoken user input by performing operationsincluding: receiving, by the one or more processors, a plurality of userfeedback items and a plurality of support solutions to the plurality ofuser feedback items; filtering, by the one or more processors, theplurality of user feedback items using a domain-specific dictionary ofterms that is derived from documents corresponding to the plurality ofsupport solutions; assigning, by one or more processors, a termfrequency-inverse document frequency (TF-IDF) score to each of thefiltered user feedback items; and generating, by the one or moreprocessors based on the TF-IDF scores, a mapping of a given phrase fromthe filtered user feedback items to one or more support solutions fromthe plurality of support solutions; and generating, by the one or moreprocessors, the items-solutions model using the mapping of the givenphrase to the one or more support solutions; and delivering, by the oneor more processors responsive to receipt of the spoken user input, theone or more determined support solution responses to a user.
 3. Themethod of claim 2, wherein delivering the one or more determined supportsolution responses comprises delivering the one or more determinedsupport solution responses as audio information that is output by aspeaker of the personal digital assistant.
 4. The method of claim 3,wherein the audio information includes recorded sound.
 5. The method ofclaim 2, wherein delivering the one or more determined support solutionresponses comprises delivering a support solution response through asmartphone.
 6. The method of claim 5, wherein delivering a supportsolution response through a smartphone includes delivering audioinformation through a telephone call on the smartphone.
 7. The method ofclaim 5, wherein delivering a support solution response through thesmartphone includes delivering audio information through an applicationon the smartphone.
 8. The method of claim 2, wherein receiving thespoken user input comprises receiving the spoken user input by way of atelephone call.
 9. A system comprising: a repository storing adomain-specific dictionary; an association engine including one or moreprocessors that perform operations including: receiving a plurality ofuser feedback items and a plurality of support solutions to theplurality of user feedback items; filtering the plurality of userfeedback items using the domain-specific dictionary of terms that isderived from documents corresponding to the plurality of supportsolutions; assigning a term frequency-inverse document frequency(TF-IDF) score to each of the filtered user feedback items; andgenerating, based on the TF-IDF scores, a mapping of a given phrase fromthe filtered user feedback items to one or more support solutions fromthe plurality of support solutions; and a solution engine including oneor more processors that perform operations including: receiving, throughan audio codec of a personal digital assistant, spoken user input;automatically determining, using the items-solutions model and based ona phrase included in the spoken user input, one or more supportsolutions responses to the spoken user input; and delivering, responsiveto receipt of the spoken user input, the one or more determined supportsolution responses to a user.
 10. The system of claim 9, whereindelivering the one or more determined support solution responsescomprises delivering the one or more determined support solutionresponses as audio information that is output by a speaker of thepersonal digital assistant.
 11. The system of claim 10, wherein theaudio information includes recorded sound.
 12. The system of claim 9,wherein delivering the one or more determined support solution responsescomprises delivering a support solution response through a smartphone.13. The system of claim 12, wherein delivering a support solutionresponse through a smartphone includes delivering audio informationthrough a telephone call on the smartphone.
 14. The system of claim 12,wherein delivering a support solution response through the smartphoneincludes delivering audio information through an application on thesmartphone.
 15. The system of claim 9, wherein receiving the spoken userinput comprises receiving the spoken user input by way of a telephonecall.
 16. A non-transitory computer storage medium encoded with acomputer program, the program comprising instructions that when executedby one or more data processing apparatus cause the data processingapparatus to perform operations comprising: receiving, through an audiocodec of a personal digital assistant, spoken user input; automaticallydetermining using an items-solutions model and based on a phraseincluded in the spoken user input, one or more support solutionresponses to the spoken user input, wherein the items-solution model isgenerated prior to receiving the spoken user input by performingoperations including: receiving a plurality of user feedback items and aplurality of support solutions to the plurality of user feedback items;filtering the plurality of user feedback items using a domain-specificdictionary of terms that is derived from documents corresponding to theplurality of support solutions; assigning a term frequency-inversedocument frequency (TF-IDF) score to each of the filtered user feedbackitems; and generating, based on the TF-IDF scores, a mapping of a givenphrase from the filtered user feedback items to one or more supportsolutions from the plurality of support solutions; and generating theitems-solutions model using the mapping of the given phrase to the oneor more support solutions; and delivering, responsive to receipt of thespoken user input, the one or more determined support solution responsesto a user.
 17. The computer storage medium of claim 16, whereindelivering the one or more determined support solution responsescomprises delivering the one or more determined support solutionresponses as audio information that is output by a speaker of thepersonal digital assistant.
 18. The computer storage medium of claim 16,wherein delivering the one or more determined support solution responsescomprises delivering a support solution response through a smartphone.19. The computer storage medium of claim 18, wherein delivering asupport solution response through a smartphone includes delivering audioinformation through a telephone call on the smartphone.
 20. The computerstorage medium of claim 18, wherein delivering a support solutionresponse through the smartphone includes delivering audio informationthrough an application on the smartphone.
 21. The computer storagemedium of claim 16, wherein receiving the spoken user input comprisesreceiving the spoken user input by way of a telephone call.